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Abstract 

A planetary rover must be able to identify states 
where it should stop or change its plan. With limited 
and infrequent communication from ground , the rover 
must recognize states accurately. However , the sensor 
data is inherently noisy ; so identifying the temporal 
patterns of data that correspond to interesting or im- 
portant states becomes a complex problem. In this pa- 
per, we present an approach to state identification us- 
ing second- order Hidden Markov Models. Models are 
trained automatically on a set of labeled training data; 
the rover uses those models to identify its state from 
the observed data. The approach is demonstrated on 
data from a planetary rover platform . 

1 Introduction 

An autonomous mobile robot exploring or operat- 
ing in an unknown environment needs to correctly 
identify fault states and environmental states in or- 
der to react to them appropriately. In the case of 
limited and delayed communication, such as for plan- 
etary rovers, human interaction is restricted, so these 
states cam only be known through interpretation of the 
sensor information on board. Some states can only be 
identified by considering a temporal sequence of sensor 
information, not simply a snapshot. Additionally, the 
sensor data may be noisy, so simple descriptions of the 
sensor data (e.g., “current rising, steady, then falling”) 
may not directly correspond to the actual data. 

Consider a planetary rover that traverses long dis- 
tances over largely unknown terrain. If it ever rolls 
over a rock, it may want to turn its cameras and take 
a picture of the newly exposed rock surface abraded by 
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the wheels. If it climbs a hill, it may be a good time to 
take images for localization and science site identifica- 
tion. Predicting the exact moment when these states 
occur is difficult or impossible for long traverses. 

An autonomous robot also needs to accurately iden- 
tify its internal fault conditions to know whether it 
should stop its activity or continue. For example, a 
rock wedged in a rover’s drive train may be a recov- 
erable error, but if left too long it may cause motor 
damage; on the other hand, frequent false alarms will 
decrease the overall effectiveness of the robot. 

This paper describes a statistical approach to robot 
state identification, using Hidden Markov Models 
(HMMs). From a set of training data, the robot builds 
models that represent the statistical properties of the 
observations corresponding to each of a set of states. 
Then, at recognition time, the robot chooses the states 
whose models best approximate the observations. 

The HMM approach is a flexible method for han- 
dling the large variability of complex temporal signals; 
for example, it is a standard method for speech recog- 
nition [7]. In contrast to dynamic time warping where 
heuristic training methods for estimating templates 
are used, stochastic modeling allows probabilistic and 
automatic training for estimating models. 

The particular approach we use is the second-order 
HMM (HMM2). HMM2s have been shown to be effi- 
cient models to capture temporal variations in speech 
[5], in many cases surpassing first-order HMMs when 
the trajectory in the state space has to be accounted 
for. HMM2s have also been applied to mobile robot 
place recognition in indoor environments [1]. 

This paper is organized as follow. We first report 
related work. In section 3, we briefly present our mo- 
bile robot. In section 4, we define the HMM2 and 
describe the algorithms used for training and recogni- 
tion. Section 5 is the description of our methodology. 
We discuss results in section 6 and give some conclu- 
sions and perspectives in section 7. 




Figure 1: The Marsokhod rover. 


2 Related Work 

A variety of approaches to state estimation and 
fault diagnosis have been proposed in the control sys- 
tems, artificial intelligence, and robotics literature. 

Techniques for state estimation of continuous val- 
ues, such as Kalman filters, can track multiple possi- 
ble hypotheses [8, 10]. However, they must be given 
an a priori model of each possible state. One of the 
strengths of the approach presented in this paper is 
its ability to construct models from training data and 
then use them for state identification. 

Qualitative model- based diagnosis techniques [2, 6] 
consider a snapshot of the system rather than its his- 
tory. In addition, the system state is assumed to be 
consistent with a propositional description of one of a 
set of possible states. The presence of noisy data and 
temporal patterns negates these assumptions. 

Hidden Markov Models have been applied to fault 
detection in continuous processes [9]; the model struc- 
ture is supplied, with only the transition probabilities 
learned from data. In the approach in this paper, the 
HMM learns without prior knowledge of the models. 

3 Marsokhod rover 

The rover used in these experiments is a Marsokhod 
rover (see figure 1), a medium-sized planetary rover 
originally developed for the Russian Mars exploration 
program; in the NASA Marsokhod, the instruments 
and electronics have been changed from the original. 


The rover has six wheels, independently driven 1 , with 
three chassis segments that articulate independently. 
It is configured with imaging cameras, a spectrome- 
ter, and an arm. The Marsokhod platform has been 
demonstrated at field tests from 1993-99 in Russia, 
Hawaii, and deserts of Arizona and California; the field 
tests were designed to study user interface issues, sci- 
ence instrument selection, and autonomy technologies. 

The Marsokhod is controlled either through se- 
quences or direct tele-operation. In either case the 
rover is sent discrete commands that describe motion 
in terms of translation and rotation rate and total 
time/distance. The Marsokhod is instrumented with 
sensors that measure body, arm, and pan/tilt geom- 
etry, wheel odometry and currents, and battery cur- 
rents. The sensors that are used in this paper are 
roll (angle from vertical in direction perpendicular 
to travel), pitch (angle from vertical in direction of 
travel), and motor currents in each of the 6 wheels. 

The experiments in this paper were performed in an 
outdoor “sandbox,” which is a gravel and sand area 
about 20m x 20m, with assorted rocks and some to- 
pography. This space is used to perform small-scale 
tests in a reasonable approximation of a planetary 
(Martian) environment. We distinguish between the 
small (less than approx. 15cm high) and large rocks 
(greater than approx. 15cm high). We also distinguish 
between the one large hill (approx, lm high) and the 
three small hills (0.3-0. 5m high). 

4 Second-order Hidden Markov Model 

In this section, we briefly present the HMM2 and 
the algorithms used for learning and recognition. A 
very complete tutorial on HMMs and their applica- 
tions to speech can be found in [7]. 

4.1 Definition 

In a HMM2, the underlying state sequence is a 
second-order Markov chain. Therefore, the probabil- 
ity of a transition between two states at time t depends 
on the states in which the process was at time t— 1 and 
t-2. A HMM2 A is specified by: 

• a set of states, 5; 

• a 3 dimensional matrix A over 5x5x5 

<*u* = Prob{q t = s k /q t - 1 = s jt q t -2 = «<) (1) 

= Prob(q t =Sk/qt-i = s ] ,q t -2 = s i ,q t - 3 =~) 

1 For the experiments, the right rear wheel had a broken gear, 
so it rolled passively. 



with Ylk=i a »j* = 1 for all 1 < i, j < N, where N 
is the number of states in the model and q t is the 
actual state at time t\ 

• a mixture of Gaussians associated with each 3 X € 5: 

M 

= ^2 C *rn^f(Ot\^l l rn,'Eirn)^ ( 2 ) 

m=l 

where c tm = 1 and is the input vector 

(i.e., the observation 2 ) at time t. 

The probability of the state sequence Q = q\, 
is defined as 

r 

Pro6((3) = JJ (3) 

t=3 ~ 

where n qi is the probability of state s qi at time t = 1 
and is the probability of the transition a qi 
at time t = 2. 

4.2 The Viterbi Algorithm 

Recognition of a given sequence of observations is 
performed by the Viterbi algorithm [3], which deter- 
mines the most likely state sequence given an obser- 
vation sequence. The Viterbi algorithm uses dynamic 
programming to compute the best partial state se- 
quence to time t for all states. The most likely state 
sequence q \ , ..., qj is obtained by keeping track of back 
pointers for each computation of which previous tran- 
sition leads to the maximal partial path probability. 

4.3 The Baum- Welch Algorithm 

Model learning is performed using the maximum 
likelihood estimation criterion that determines the 
best model parameters according to the corpus of 
items. It must be noted that this criterion does not 
try to separate models like a neural network, but only 
tries to increase the probability that a model generates 
its corpus independently of what the other models can 
do. Intuitively, this algorithm counts the number of 
occurrences of each transition between the states in 
the training corpus. Each count is weighted by the 
probability of the alignment (state, observation). 

5 Application to rover state identifica- 
tion 

To allow the Marsokhod to learn to recognize par- 
ticular situations while it is exploring the “sandbox” , 

2 An observation is defined as the measure of one or several 

sensors at a given time. 



Figure 2: Topology of states used for each model of 
situation 

we have to address several major issues: defining a set 
of situations; defining a second-order hidden Markov 
model associated with each situation; collecting a cor- 
pus of observations during several runs and labeling 
this corpus by finding temporal borders of each situa- 
tion that the robot has observed during its run. 

5.1 The set of situations 

Currently, we model six distinct situations that are 
representative of a typical outdoor exploration envi- 
ronment: when the robot is climbing a small rock on 
its left (resp. right) side, a big rock on its left side 3 , a 
small (resp. big) hill, and a default situation of level 
ground. These situations are chosen for the fact that 
they are repeatable and human-observable (for label- 
ing); internal fault identification would have required 
instrumenting the rover to cause faults on command, 
which is not currently possible on the Marsokhod. 

We make the hypothesis that two or more of these 
situations cannot overlap (e.g., a small rock on the 
right side while climbing a big hill). This set of items 
is a complete description of what the mobile robot can 
see during its runs. All other unforeseen situations, 
like flat rocks or holes, are treated as noise. 

5.2 The model to represent each situation 

In the formalism described in section 4.1, each sit- 
uation to be recognized is modeled by a HMM2 whose 
topology is depicted in figure 2. This topology is well 
suited for the type of recognition we want to perform. 
Recognition begins in the leftmost state, and each time 
am event characterizing the situation is recognized it 
advances to the right. When the rightmost state has 
been reached, the recognition of the situation is com- 
plete. The five-state model has been chosen experi- . 
mentally to give the best rate of recognition. 

5.3 Corpus collecting and labeling 

We built six corpora to traun a model for each situ- 
ation. For this, our mobile robot made approximately 

3 The situation of a big rock on the right side was not con- 
sidered because of the non-functional right-side wheel. 




Figure 3: Segmentation and labeling of a run. 


fifty runs in the sandbox. For each run, the robot re- 
ceived one discrete translation command ranging from 
three meters to twenty meters. Rotation motions are 
not part of the corpus. Each run contains different 
situations, but each run is unique (i.e., the area tra- 
versed and the sequence of situations during the run is 
different each time) . A run contains not only one sit- 
uation but ail the situations seen while running. For 
each run, we noted the situations seen during the run, 
for later segmentation and labeling purposes. 

As hidden Markov models have the ability to model 
signals whose properties change with time, we have to 
choose a set of sensors (as the observation) that have 
noticeable variations when the Marsokhod is crossing a 
rock or a hill. From the sensors described in section 3, 
we identified eight such sensors: roll, pitch, and the six 
wheel currents. We define coarse rules to identify each 
situation (for segmentation and labeling by humans): 

• When the robot crosses a small (resp. big) rock on 
its left, we notice a distinct sensor pattern. In all 
cases, the roll sensor shows a small (resp. big) in- 
crease when climbing the rock, then a small (resp. 
big), sudden decrease when descending from the 
rock. These two variations usually appear sequen- 
tially on the front, middle, and rear left wheels. The 
pitch sensor always shows a small (resp. big) in- 
crease, then a small (resp. big), sudden decrease, 
and finally a small (resp. big) increase. There is 
little variation on the right wheels. 

• When the robot crosses a small rock on its right 
side, we observe variations symmetric to the case of 
a small rock on the left side. 


• When the robot crosses a small (resp. big) hill, the 
pitch sensor usually shows a small (resp. big) in- 
crease, then a small (resp. big) decrease, and fi- 
nally a small (resp. big) increase. There is not 
always variation in the roll sensor. However, there 
is a gradual, small (resp. big) increase followed by a 
gradual, small (resp. big) decrease on all (or almost 
all) the six wheel current sensors. 

These rules are used to segment and label each run. 
An example of segmentation and labeling is given in 
figure 3. The sensors are in the following order (from 
the top): roil, pitch, the three left wheel currents, and 
the three right wheel currents. A vertical line marks 
the beginning or the end of a situation. The default 
situation alternates with the other situations. The 
sequence of situations in the figure is the following 
(as labeled in the figure): small rock on the left side, 
default situation, big rock on the right side, default 
situation, small hill, default situation, and big hill. 

5.4 Model training 

As we want to compare different possibilities, we 
perform a separate model training for each of three 
sets of input data. The observation used as input of 
each model to train consists of: 

• eight coefficients: the first derivative (i.e., the vari- 
ation) of the values of the eight sensors used for 
segmentation. 

• six coefficients: the first derivative (i.e., the varia- 
tion) of the values of the six wheel current sensors. 

• two coefficients: the first derivative (i.e., the varia- 
tion) of the values of the roll and the pitch sensors. 

Each training uses segmented data, and each model is 
trained independently with its corpus. 

The three different model trainings are performed 
for two reasons. First is to check whether the eight 
sensors used for the segmentation are necessary to 
learn and recognize situations, or whether a subset 
set is sufficient. Second, we want to be able to rec- 
ognize situations even if one or more sensors do not 
work; e.g., if some wheel sensors do not work it will 
affect (during recognition) the models using the six 
wheel current sensors or the eight sensors but not the 
models using just the roll and the pitch sensor. 

5.5 The recognition phase 

The robot’s environment is described using a gram- 
mar that accepts only certain sequences of models. In 
this grammar, all the HMM2s are chained in a bigger 
HMM on which the Viterbi algorithm is used. The 



best sequence of states determines the ordered list of 
situations seen during the run. The list of situations 
is known definitively when the run is completed. 

The goal of recognition is to spot the five situations 
(small rock on the left or right; big rock on the left; 
small or big hill) while the robot moves in the sandbox. 
The default situation model connects two items much 
like silence between two words in speech recognition. 

During the recognition phase, the robot was oper- 
ated as for corpus collecting. We took approximately 
40 acquisitions and used the six trained models to per- 
form the recognition. A situation is recognized if it is 
detected by the corresponding model close to its real 
geometric position. A few types of errors can occur: 

Insertions: the robot sees a non-existing situation. 
Deletions: the robot misses a situation; 
Substitutions: the robot confuses two situations. 

We perform three independent recognitions (corre- 
sponding to the three learning situations). 

6 Results and discussion 

The results are presented first as confusion ma- 
trices, where an element Cij is the number of times 
the model j has been recognized when the right an- 
swer was situation i, and second with the global rate 
of recognition, substitution, omission and insertion. 

In each confusion matrix, the acronyms used are: 
BL = big rock on the left, SL = small rock on the 
left, SR = small rock on the right, BH = big hill, and 
SH = small hill. 

6.1 Experiment with eight sensors 



BL 

SL 

SR 

BH 

SH 

Ins 

BL 

19 

3 

1 

- 

- 

9 

SL 

3 

25 

- 

- 

- 

12 

SR 

1 

2 

31 

- 

1 

26 

BH 

1 

- 

- 

20 

2 

15 

SH 

- 

- 

- 

1 

23 

28 

Omi 

1 

1 

- 

- 

- 

- 

Total 

25 

31 

32 

21 

26 

90 

Reco 

76% 


97% 

~95^“ 




Table 1: Confusion matrix of situations, eight sensors. 


For eight sensors, as each situation can be easily 
distinguished from the others, the global rate of recog- 
nition is excellent (87%) (see tables 1,2). Small (resp. 
big) rocks on the left are sometimes confused with big 
(resp. small) rocks on the left; the signal provided 



8 sensors 

6 sensors 

2 sensors 


# 

% 

# 

% 

# 

% 

Seen 

135 

100 

135 

100 

135 

100 

Recognized 

118 

87 

113 

84 

82 

61 

Substituted 

15 

11 

21 

15 

50 

37 

Omitted 

2 

2 

1 

1 

3 

2 

Inserted 

90 

67 

124 

92 

42 

31 


Table 2: Global rate of recognition, comparing 8 sen- 
sors, 6 sensors, and 2 sensors. 


by the sensors does not contain the information for 
discriminating these two models. In fact, the varia- 
tions on the sensors are nearly the same. The only 
criterion which distinguishes these two models is the 
amplitude of the variation on the three left wheels, 
and visibly it is not sufficient. The small rocks on the 
right are perfectly recognized. This situation has a 
very distinctive pattern, and only with difficulty can 
it be confused with another. The fact that we could 
not learn and recognize a situation where the robot is 
crossing a big rock on its right avoids any confusion. 

The major problem is the high rate of insertion. 
This rate is due to the noise of the sensors being rec- 
ognized as a situation. This is especially the case for 
situations characterized only by small variations on a 
part (or all) of the set of sensors, in particular the 
crossing of a small hill. 

6.2 Experiment with six sensors 



BL 

SL 

SR 

BH 

SH 
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BL 

17 
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- 

- 

10 

SL 

4 

24 

2 

- 

1 
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SR 

3 

- 

29 

- 

1 

44 

BH 

- 

1 

- 

20 

1 

19 

SH 

1 

- 

- 

1 

23 

32 

Omi 

- 

1 

- 

- 

- 

- 

Total 

25 

31 

32 

21 

26 

124 

Reco 

68% 

77% 

91% 

95% 

88% 



Table 3: Confusion matrix of situations, six sensors. 


With six sensors, the global rate of recognition is 
still very good (see tables 3, 2). There is only four 
more percent of substitutions due to the loss of infor- 
mation used to distinguish situations. On the other 
hand, the rate of insertion increased by 25%. With 
only the six wheel current sensors, nearly one recogni- 
tion out of two is an insertion. The six wheel current 
sensors are very noisy, and the roll and pitch sensors 






are useful to distinguish between simple noise and real 
situations. This explains the increase of the insertions. 

6.3 Experiment with two sensors 
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25 

31 

32 
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26 
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60% 

55% 

84% 

~6f%~ 

~35%~ 



Table 4: Confusion matrix of situations, two sensors. 


With only the roll and pitch sensors, the global rate 
of recognition remains good, and the rate of insertions 
significantly decreases (see tables 4, 2). In fact, these 
two sensors are not too noisy, and when there is a vari- 
ation on these sensors it generally corresponds to a real 
situation. But these two sensors do not provide suf- 
ficient information to distinguish between situations, 
which is why there is a high rate of substitution. 

7 Conclusion and perspectives 

In this paper, we have presented a new method to 
learn and recognize situations in an outdoor environ- 
ment with second-order hidden Markov models. One 
of the main interests of this work is the specification 
of an automatic learning algorithm of the environment 
that allows recognition of typical situations. 

This method gives very good results, and has a good 
robustness to noise, verifying that HMM2s are well 
suited to learn and recognize situations during explo- 
ration of an outdoor environment. Prom the results of 
experiments, we can draw some conclusions. The best 
way to perform recognition is with eight sensors: the 
rate of recognition is a little bit better than for six sen- 
sors and the rate of insertion is very smaller. This can 
be explained by the fact that the six wheels current 
sensors are very noisy, and the use of the roll and pitch 
sensors, which are not too noisy, can distinguish be- 
tween a situation to recognize and a simple noise on 
the current wheel sensors. Nonetheless, the learned 
models in the two last experiments can be useful in 
long exploration where sensors can fail. 

Our method can be extended to fault detection, for 
example broken wheels or sensor failure. In fact, we 


can build one model of a particular situation where 
all sensors work and several models of this situation 
where one or several sensors are broken: for example a 
model of a big rock on the right side and a model of a 
big rock on the right when the front left wheel is bro- 
ken. Using these models, we can recognize situations 
associated with the state of the sensors of the robot. 

To recognize situations online during exploration, 
we can use a varient of the Viterbi algorithm called 
Viterbi-block [4]. This algorithm is based on a lo- 
cal optimum comparison of the different probabili- 
ties computed by the Viterbi algorithm during time- 
warping of a shift-window of fixed length in the signal 
and the different HMMs. This algorithm can recognize 
situations a few meters after they have been seen. 
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